SYM-AM-1 6-031 


PROCEEDINGS 

OF  THE 

THIRTEENTH  ANNUAL 
ACQUISITION  RESEARCH 
SYMPOSIUM 


WEDNESDAY  SESSIONS 
VOLUME  I 


Further  Evidence  on  the  Effect  of  Acquisition  Policy  and 

Process  on  Cost  Growth 

David  McNicol,  Research  Staff  Member,  Institute  for  Defense  Analyses 
David  Tate,  Research  Staff  Member,  Institute  for  Defense  Analyses 

Published  April  30,  2016 


Approved  for  public  release;  distribution  is  unlimited. 
Prepared  for  the  Naval  Postgraduate  School,  Monterey,  CA  93943. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


The  research  presented  in  this  report  was  supported  by  the  Acquisition  Research 
Program  of  the  Graduate  School  of  Business  &  Public  Policy  at  the  Naval 
Postgraduate  School. 

To  request  defense  acquisition  research,  to  become  a  research  sponsor,  or  to  print 
additional  copies  of  reports,  please  contact  any  of  the  staff  listed  on  the  Acquisition 
Research  Program  website  (www.acquisitionresearch.net). 


\i  mnusmtitM 'i vriAn r 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Panel  5.  Quantitative  Analyses  of  Acquisition 
Outcome  Drivers 


Wednesday,  May  4,  2016 

1:45  p.m.  - 
3:15  p.m. 

Chair:  William  Gates,  Dean,  Graduate  School  of  Business  and  Public  Policy, 

NPS 

Consequences  ofBBP’s  Affordability  Initiative 

Gregory  Davis,  Research  Staff  Member,  Institute  for  Defense  Analyses 
Lawrence  Goeller,  Defense  Acquisition  Analyst,  Institute  for  Defense 
Analyses 

Stanley  Horowitz,  Assistant  Director,  Cost  Analysis  and  Research 

Division,  Institute  for  Defense  Analyses 

Further  Evidence  on  the  Effect  of  Acquisition  Policy  and  Process  on  Cost 
Growth 

David  McNicol,  Research  Staff  Member,  Institute  for  Defense  Analyses 
David  Tate,  Research  Staff  Member,  Institute  for  Defense  Analyses 

Preparing  to  Be  Wrong 

Prashant  Patel,  Research  Staff  Member,  Institute  for  Defense  Analyses 
Michael  Fischerkeller,  Research  Staff  Member,  Institute  for  Defense 
Analyses 

ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-117- 


Further  Evidence  on  the  Effect  of  Acquisition  Policy  and 

Process  on  Cost  Growth 


David  L.  McNicol — joined  the  Department  of  Defense  (DoD)  in  1982.  From  1988  until  2002,  he  was 
a  Deputy  Director  of  Program  Analysis  and  Evaluation  (PA&E).  Earlier,  Dr.  McNicol  taught  at  the 
University  of  Pennsylvania  and  the  California  Institute  of  Technology.  He  holds  a  BA  in  economics 
from  Harvard  and  an  MS  (management)  and  PhD  (economics  and  finance)  from  MIT.  Employed  at 
the  Institute  for  Defense  Analyses  (IDA)  since  his  retirement  from  the  DoD,  he  became  director  of  the 
Cost  Analysis  and  Research  Division  in  2006.  Still  at  IDA,  Dr.  McNicol  stepped  down  in  2012  to  return 
to  his  previous  role  as  a  Research  Staff  Member,  [dmcnicol@ida.org] 

David  M.  Tate — joined  the  research  staff  of  the  Institute  for  Defense  Analyses’  Cost  Analysis  and 
Research  Division  in  2000.  Since  then,  he  has  worked  on  a  wide  variety  of  resource  analysis  and 
quantitative  modeling  projects  related  to  national  security.  These  include  an  independent  cost 
estimate  of  Future  Combat  Systems  development  costs,  investigation  of  apparent  inequities  in 
Veterans’  Disability  Benefit  adjudications,  and  modeling  and  optimization  of  resource-constrained 
acquisition  portfolios.  Dr.  Tate  holds  bachelor’s  degrees  in  philosophy  and  mathematical  sciences 
from  the  Johns  Hopkins  University,  and  MS  and  PhD  degrees  in  operations  research  from  Cornell 
University,  [dtate@ida.org] 

Abstract 

Institute  for  Defense  Analyses  Paper  P-5126  found  that  additional  acquisition  reforms  after 
those  introduced  in  mid-1969  by  then  Deputy  Secretary  of  Defense  David  Packard  did  not 
significantly  reduce  cost  growth  on  Major  Defense  Acquisition  Programs  (MDAPs).  That 
conclusion — while  interesting — is  incomplete,  as  it  leaves  open  the  possibility  that  the 
Packard  reforms  reduced  cost  growth  compared  to  the  record  of  the  1960s,  which  is  the 
issue  examined  in  this  paper.  The  paper  finds  that  average  cost  growth  of  MDAPs  that 
entered  Engineering  and  Manufacturing  Development  during  fiscal  year  (FY)  1970-FY  1980 
was  significantly  lower  than  the  average  of  those  that  entered  during  FY  1964-FY  1969.  It 
also  probably  was  significantly  lower  than  the  average  during  FY  1994-FY  2000  when  Office 
of  the  Secretary  of  Defense  (OSD)-level  oversight  of  MDAPs  was  less  stringent.  These  stand 
as  instances  of  a  significant  association  between  changes  in  OSD-level  oversight  and  cost 
growth.  The  paper  also  provides  evidence  that  average  cost  growth  in  FY  1964-FY  1969  and 
FY  1994-FY  2000  was  particularly  high  largely  because  the  proportion  of  MDAPs  that 
experienced  extremely  high  cost  growth  was  significantly  larger  than  it  was  in  other  periods. 

Introduction 

McNicol  and  Wu  (2014;  hereafter  referred  to  as  P-5126)  reported  two  significant 
findings.  First,  Major  Defense  Acquisition  Programs  (MDAPs)  that  entered  Engineering  and 
Manufacturing  Development  (EMD)  during  “bust”  funding  climates  on  average  had  much 
higher  cost  growth  than  those  that  entered  EMD  during  “boom”  climates.  Second,  the  paper 
found  that  additional  reforms  after  those  introduced  in  mid-1969  by  then  Deputy  Secretary  of 
Defense  David  Packard  had  not  significantly  reduced  cost  growth. 

As  P-5126  noted,  the  latter  conclusion  leaves  open  the  possibility  that  the  Packard 
reforms  reduced  cost  growth  compared  to  the  record  of  the  1960s.  If  in  fact  they  did,  the 
conclusion  of  that  paper  would  have  to  be  amended  to  read:  The  introduction  in  1969  of 
effective  Office  of  the  Secretary  of  Defense  (OSD)-level  oversight  of  major  acquisition 
programs  reduced  cost  growth,  but  the  additional  reforms  of  the  1970s,  1980s,  and  early 
1990s  did  not  result  in  further  reductions.  Along  the  same  line,  it  is  of  interest  to  revisit  the 
mixed  evidence  P-5126  found  on  the  effect  on  cost  growth  of  less  active  OSD-level 
oversight  of  1994-2000.  The  crucial  question  is  whether  there  is  statistical  evidence  that 
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cost  growth  decreased  when  OSD-level  controls  were  imposed  and  also  increased  when 
those  controls  were  relaxed. 

This  is  not  simply  an  historical  question,  because  the  main  features  of  today’s  OSD 
level  acquisition  oversight  process  remain  those  of  the  process  installed  by  Packard  in  mid- 
1969.  Moreover,  the  issue  is  salient  now  because  of  its  implications  for  ongoing  discussions 
of  reform  of  the  DoD  weapon  system  acquisition  process. 

The  database  available  for  P-5126  did  not  contain  cost  growth  estimates  for  any 
MDAPs  that  entered  EMD  during  the  1960s,  so  that  paper  could  not  compare  cost  growth 
pre-  and  post-Packard.  This  paper  uses  cost  growth  data  for  programs  that  entered  EMD  in 
the  1960s  from  two  previous  studies  (Jarvaise,  Drezner,  &  Norton,  1996;  Tyson,  Om, 
Gogerty,  &  Nelson,  1992).  It  also  uses  a  different  cost  growth  metric  and  employs  additional 
statistical  tests. 

The  next  section  briefly  describes  the  OSD-level  acquisition  oversight  introduced  by 
Robert  McNamara  in  the  mid-1960s  and  the  changes  made  to  it  in  1969  by  Packard.  It  is 
necessary  to  do  this  because  the  McNamara  reforms  are  no  longer  part  of  the  collective 
memory  of  the  DoD  acquisition  community.  Subsequent  sections  then  turn  to  the  statistical 
analysis  and  the  conclusions  it  suggests.  These  sections  assume  that  the  reader  has  a 
working  familiarity  with  acquisition  process  and  policies.  Those  who  do  not  may  wish  to 
consult  Fox  (201 1 ).  Readers  who  want  a  more  detailed  understanding  of  the  data  used  and 
the  way  they  were  binned  should  consult  Appendixes  A  and  B  of  McNicol,  Tate,  Burns,  and 
Wu  (2016). 

Origins  of  the  OSD-Level  Acquisition  Oversight  Process 

From  the  creation  of  the  National  Security  Establishment  in  1947  through  1960,  the 
OSD  had  no  institutionalized  process  for  the  oversight  of  major  weapon  system 
acquisitions.1  The  origins  of  the  OSD-level  process  for  overseeing  major  weapon  system 
acquisitions  lie  in  initiatives  taken  by  McNamara,  of  which  the  following  are  especially 
relevant  for  current  purposes: 

•  Promulgation  of  policy  on  contract  types 

•  Establishment  of  milestone  decision  points  and  the  Development  Concept 
Paper  (DCP) 

•  Active  oversight  of  ongoing  MDAPs2 


1  The  Secretary  of  Defense  could,  and  on  occasion  did,  act  to  cancel  or  initiate  major  acquisitions. 
Major  acquisition  programs  were  also  subject  to  review  during  the  budget  cycle  by  the  Office  of  the 
Assistant  Secretary  of  Defense  (Comptroller)  and  the  Office  of  Management  and  Budget.  Additionally, 
a  major  building  block  of  McNamara’s  process  began  operating  in  1959.  See  O’Neil  and  Porter  (2011, 
p.  25). 

2  These  categories  are  abstracted  from  Fox  (2011,  p.  35-45).  Fox  also  notes  that  McNamara  moved 
to  consolidate  acquisition  functions  in  defense  agencies — e.g.,  the  agency  that  became  the  Defense 
Logistics  Agency — and  promoted  the  use  by  program  managers  of  particular  management  tools  such 
as  PERT  and  earned  value.  In  addition,  there  are  several  cases — most  notably  the  F-1 1 1 — in  which 
McNamara  played  a  very  active  role  in  the  oversight  of  the  program.  These  cases  almost  certainly  are 
exceptions,  but  the  literature  survey  done  for  this  paper  uncovered  little  about  how  the  process 
worked  in  the  more  typical  cases.  Adding  to  the  confusion,  the  sources  consulted  suggest  that  during 
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These  initiatives  were  an  embryonic  OSD-level  acquisition  oversight  process. 

McNamara  directed  the  use  of  Total  Package  Procurement  (TPP)  when  it  was 
judged  to  be  practicable  and,  when  not,  Fixed  Price  Incentive  Fee  (FPIF)  or  Cost  Plus 
Incentive  Fee  (CPIF)  contracts.* * 3  By  1966,  McNamara  had  concluded  that  TPP  contracts 
were  in  fact  not  a  practicable  way  to  acquire  major  weapon  systems,  although  acquisition 
policy  apparently  still  had  a  tilt  towards  fixed  price  contracts,  even  for  development.  Packard 
picked  up  on  this  topic  where  McNamara  left  off.  He  ruled  out  the  use  of  TPP  and 
discouraged  the  use  of  FPIF  for  development  contracts  in  favor  of  CPIF.  (Cost  Plus  Award 
Fee  may  not  have  been  included  in  the  contracting  play  book  yet.)  As  a  general  matter, 
Packard’s  policy  was  to  match  contract  terms  to  the  riskiness  of  the  acquisition. 

Packard’s  establishment  of  the  Defense  Systems  Acquisition  Review  Council 
(DSARC)  often  is  seen  as  the  hallmark  of  his  1969  reforms.  The  notion  of  milestone  reviews, 
however,  entered  the  OSD-level  acquisition  process  in  1964  with  issuance  of  DoD  Directive 
(DoDD)  3200.9,  Initiation  of  Engineering  and  Operational  Systems  Development.4  This 
original  version  of  the  directive  set  one  point  at  which  OSD — in  principle,  the  Secretary  of 
Defense — approval  was  required  for  an  acquisition  program  to  proceed.  In  1965,  a  second 
decision  point  was  added,  and  the  Director,  Defense  Research  and  Engineering  (DDR&E), 
instituted  the  precursor  of  the  DCP,  which,  starting  in  1968,  was  required  to  initiate  any 
major  development  project.  DDR&E  coordinated  initial  DCPs  with  concerned  OSD  offices 
(and  probably  the  Joint  Staff  and  other  Services;  O’Neil  &  Porter,  201 1 )  and  acted  as  what 
now  would  be  called  the  Milestone  Decision  Authority  (MDA)  for  the  initial  DCP  (Borklund, 
1969).  Once  approved  by  DDR&E,  the  proposed  new  start  went  to  the  Secretary  of 
Defense,  although  the  sources  consulted  do  not  indicate  whether  it  went  as  a  separate 
action  or  as  part  of  the  Service’s  budget  submission.  It  is  also  not  clear  which  OSD  official 
was  the  MDA  for  the  second  milestone. 

Viewed  against  this  background,  the  establishment  of  the  DSARC  was  an 
evolutionary  step.  The  Development  Concept  Paper  was  renamed  the  Decision 
Coordinating  Paper  (retaining  the  acronym)  to  reflect  the  broader  scope  of  the  new 
milestone  definitions.  The  MDA  at  Milestone  (MS)  I  and  MS  II  was  DDR&E;  the  Assistant 
Secretary  of  Defense  for  Installations  and  Logistics  was  the  MDA  for  MS  III.  Decisions  at  the 
DSARC  level  were  advisory  to  the  Secretary  and  Deputy  Secretary  of  Defense  but,  apart 
from  exceptional  cases,  they  probably  reached  that  level  by  way  of  the  Service’s  proposed 
budgets  (and  the  Comptroller  was  the  backstop  enforcer  of  the  requirement  for  milestone 
approval  before  a  program  could  advance  to  the  next  stage). 

The  OSD  had  a  much  larger  role  in  oversight  of  major  acquisition  programs  under 
the  DSARC  process  than  it  did  pre-1961.  The  picture  in  contrast  to  the  McNamara  years  is 


the  McNamara  years  a  major  acquisition  program  might  arise  in  either  the  Planning,  Programming, 

and  Budgeting  System  (PPBS)  or  in  the  acquisition  process. 

3  Fox  (201 1,  p.  38),  following  Adams,  Murphy,  and  Rosenau  (1983,  pp.  19-20).  A  TPP  contract  is  one 
that  covers  EMD,  at  least  a  significant  portion  of  procurement,  and  at  least  part  of  the  support  of  the 
system  (e.g.,  depot  maintenance). 

4  The  first  version  of  DoDD  3200.9  was  issued  in  1964.  A  revision  that  made  provision  for  the 
Contract  Definition  Phase  was  issued  July  1,  1965.  See  Glennan  (1965,  p.  12).  O’Neil  and  Porter 
(2011,  pp.  25-47)  sketch  how  the  process  evolved  and  worked  during  the  1960s. 
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less  clear-cut.  On  one  hand,  under  the  new  acquisition  directives,  the  Secretary  of  Defense, 
while  retaining  full  legal  authority  over  acquisition  programs,  would  act  through  the 
established  acquisition  process  except  in  extraordinary  circumstances,  which  in  comparison 
to  cases  such  as  the  F-1 1 1  implied  less  OSD-level  control  over  acquisitions.  On  the  other 
hand,  the  DSARC  had  a  greater  substantive  scope  for  the  more  typical  program  and  was 
more  tightly  organized.  For  the  large  majority  of  major  acquisition  programs,  then,  the  new 
DSARC  process  probably  was  more  effective.5 

The  most  consequential  of  Packard’s  1969  reforms  involved  the  substance  of  the 
milestones.6  The  1965  version  of  the  DoDD  3200.9  process  had  three  phases.  The  first  of 
these  “was  called  concept  formulation.  During  concept  formulation  OSD  and  the  Service(s) 
involved  assured  themselves  that  they  were  buying  the  right  system  to  meet  real  needs  and 
that  the  technology  was  fully  ready”  (O’Neil  &  Porter,  201 1 ,  p.  30).  Concept  formulation 
typically  was  initiated  by  a  Service  but  involved  DDR&E  and  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Systems  Analysis  (OASD[SAj),  and  included  what  would  now  be 
called  an  Analysis  of  Alternatives  led  by  OASD(SA).  It  also  apparently  included  what  would 
later  be  called  a  Mission  Element  Need  Statement  as  well  as  the  main  parts  of  an 
Acquisition  Strategy  and  plans  for  oversight  of  the  program  as  it  proceeded. 

Approval  to  proceed  from  the  Concept  Formulation  phase  authorized  the  Service 
sponsoring  the  program  to  fund  at  least  one  company  to  prepare  a  definitized  contract 
proposal.  The  OSD  (milestone)  review  of  these  proposals  was  the  basis  for  award  of  a 
contract,  usually  to  a  single  source,  for  development  and  procurement  of  the  system.  That  is 
to  say,  the  second  of  DoDD  3200. 9’s  milestones  combined  what  now  would  be  called  MS  B 
and  MS  C  authority. 

Packard’s  reforms  separated  the  decision  to  allow  the  program  to  enter  EMD  from 
the  decision  to  enter  the  production  phase  (now  MS  C)  and  required  OSD-level  approval  of 
each  decision.  Packard  also  established  a  new  Validation  Phase,  which  has  at  various  times 
since  been  called  Demonstration  and  Validation,  Program  Development  and  Risk  Reduction 
and,  currently,  Technology  Maturation  and  Risk  Reduction.  MS  I  (now  MS  A)  authorized 
entry  into  this  phase.  DoDD  3200. 9’s  Contract  Definition  phase  was  collapsed  into  the  new 
and  broader  Validation  phase.  These  changes  were  more  revolutionary  than  evolutionary.7 

The  provisional  judgment  offered  here  is  that  Packard’s  acquisition  reforms  provide  a 
plausible  reason  for  expecting  program  outcomes — measured  by  cost  growth,  schedule 
slips,  and  performance  shortfalls — to  be  better  than  what  was  achieved  during  the 


5  Murdock  (1974,  pp.  155-179),  disagrees  with  this  judgment.  Murdock  is  primarily  concerned  with 
Systems  Analysis  and  resource  allocation,  but  also  comments  specifically  on  the  acquisition  process. 
In  particular,  he  notes  that  the  new  Decision  Coordinating  Paper  did  not  provide  “any  mechanism  for 
ongoing  managerial  control.”  This  is  accurate  in  that  the  Packard  reforms  placed  management  of  the 
programs  in  the  hands  of  the  Services.  It  is  incomplete  in  that  the  Services  were  responsible  for 
staying  within  what  would  later  be  called  the  Acquisition  Program  Baseline,  and  the  MDA  was 
enjoined  to  act  in  cases  in  which  they  did  not. 

6  Fox  (2011,  p.  57),  provides  a  useful  schematic  comparison  of  the  DoDD  3200.9  milestones  and 
those  of  Packard’s  DoDD  5000.1/DoDI  5000.2. 

7  DoDI  5000.2,  issued  October  23,  2000,  formally  established  MSs  A,  B,  and  C  (replacing  MSs  I,  II, 
and  III)  as  the  main  decision  points  for  an  MDAP.  The  definitions  are  such  that  MS  B  is  placed 
several  months  earlier  in  the  process  than  MS  II. 
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McNamara-Clifford  years.  This  judgment  does  not  imply  that  the  DoD  was  doing  a  better  job 
of  deciding  what  to  buy,  but  only  that,  as  a  result  of  the  Packard  reforms,  the  OSD  became 
more  effective  in  oversight  of  acquisition  programs  from  MS  II  through  the  completion  of 
procurement. 

Statistical  Analysis  of  Average  Cost  Growth 

The  statistical  analysis  presented  here  rests  on  definitions  of  periods  delimited  by 
major  changes  in  acquisition  policy  and  process.  Two  of  these  already  encountered  are 
labeled  “McNamara-Clifford”  and  “DSARC.”  Four  additional  acquisition  periods  are 
introduced  below.  Another  part  of  the  scaffolding  of  the  analysis  is  funding  climate.  Two 
climates  are  distinguished — “bust”  and  “boom.”  Three  of  the  acquisition  periods  include  both 
bust  and  boom  phases  and  three  were  entirely  in  a  single  funding  climate.  Finally,  the 
analysis  rests  on  a  set  of  conventions  concerning  which  MDAPs  are  included  in  the 
database  and  the  way  in  which  cost  growth  is  measured.  See  Appendix  A  of  McNicol  et  al. 
(2016)  for  an  explanation  of  the  basis  of  the  boundaries  separating  the  successive 
acquisition  periods  and  the  funding  climates.  Appendix  B  of  McNicol  et  al.  states  the 
conventions  used  in  assembling  the  database  and  identifies  the  sources  of  the  data  used. 

This  section  considers  whether  there  are  statistically  significant  differences  in  cost 
growth  across  the  successive  acquisition  regimes  in  bust  climates.  The  measure  of  cost 
growth  used  is  Average  Procurement  Unit  Cost  (APUC).  “APUC  growth”  means  growth  in 
APUC  in  program  base  year  dollars  normalized  to  the  baseline  quantity  approved  at  MS  B. 
Attention  in  this  section  and  most  of  the  one  that  follows  is  limited  to  MDAPs  that  entered 
EMD  during  bust  periods  because  the  interesting  findings  arise  from  the  analysis  of  those 
periods.  Results  for  boom  periods  are  briefly  mentioned  at  the  end  of  the  following  section. 

Table  1  reports  average  APUC  growth  experienced  by  MDAPs  that  passed  MS  ll/B 
during  each  of  the  six  acquisition  regimes  in  a  bust  climate.  It  is  important  to  bear  in  mind 
that  APUC  growth  is  computed  by  comparing  the  MS  ll/B  baseline  value  for  APUC — which 
can  be  thought  of  as  a  goal  or  a  prediction — to  the  actual  APUC,  normalized  to  the  MS  ll/B 
quantity8  (or,  for  ongoing  programs,  to  the  projected  APUC  in  the  December  2012  Selected 
Acquisition  Reports  [SARs],  which  were  the  most  recent  available  when  this  project  began).9 
The  APUC  growth  figures  shown  are  the  quantity  normalized  average  for  the  MDAPs  in  that 
acquisition  regime,  binned  by  the  year  the  MDAP  passed  MS  ll/B.  This  is  done  on  the 
hypothesis  that  the  acquisition  policies  and  processes  in  place  when  an  MDAP  passes  MS 
ll/B,  particularly  the  rigor  of  the  MS  ll/B  review,  have  an  effect  on  the  amount  of  cost  growth 
it  experiences  in  the  future. 


8  About  three-quarters  of  the  MDAPs  that  passed  MS  ll/B  in  the  period  FY  1988-FY  2007  acquired  at 
least  90%  of  their  MS  ll/B  baseline  quantity.  The  median  program  acquired  100%  and  the  average 
program  acquired  111%.  See  McNicol  etal.  (2015,  p.  7-8). 

9  We  follow  the  convention  of  not  including  in  the  database  any  MDAP  that  was  not  at  least  five  years 
beyond  EMD  (so  that  cost  growth  would  have  time  to  appear).  The  most  recent  SARs  available  when 
P  5126  was  written  were  those  for  December  2012.  Consequently,  MDAPs  that  passed  MS  B  during 
FY  2007  were  the  most  recent  included  in  the  database. 
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Table  1.  Average  APUC  Growth  by  Acquisition  Regime  for  MDAPs  That  Entered 

EMD  During  a  Bust  Funding  Climate 


Acquisition  Regime 

Period  (FY) 

Average  APUC 
Growth* 

McNamara-Clifford 

1964-1969 

85%  (20) 

DSARC 

1970-1980 

39%  (53) 

Post  Carlucci  DSARC 

1987-1989 

44%  (12) 

Defense  Acquisition  Board  (DAB) 

1990-1993 

32%  (11) 

Acquisition  Reform  (AR) 

1994-2000 

78%  (27) 

DAB  post  AR 

2001-2002 

113%  (6) 

Note.  Numbers  in  parentheses  are  the  number  of  observations  in  the  cell. 

*  Normalized  for  changes  in  quantity. 

A  plausible  reading  of  the  averages  in  Table  1  is  as  follows:  Packard’s  radically  new 
acquisition  phases  and  his  more  highly  structured  process  were  successful  in  reducing 
APUC  growth,  which  fell  to  less  than  half  the  average  level  it  had  during  the  1960s.  Perhaps 
encouraged  by  Packard’s  success  and  public  distaste  for  cost  growth,  acquisition  reform 
efforts  persisted,  but  had  no  appreciable  further  effect  on  average  cost  growth  prior  to  the 
AR  years.  Reduction  of  OSD  oversight  during  the  AR  era  coincided  with  the  return  of 
average  APUC  growth  to  nearly  its  1960s  level.  In  sum,  the  Packard  reforms  of  late  FY  1969 
appear  to  have  reduced  APUC  growth;  they  were  not  significantly  improved  upon  in  this 
respect  through  the  bust  years  that  followed;  and  the  AR  years  were  associated  with  higher 
APUC  growth,  which  may  be  related  to  a  reduction  of  OSD-level  oversight. 

The  question  for  the  statistical  analysis  in  an  exploratory  context  is:  Can  cause 
reasonably  be  ascribed  to  the  period-to-period  changes  in  APUC  growth,  or  are  those 
changes  more  likely  simply  random  fluctuations  in  the  data? 

It  is  useful  to  break  this  question  into  three  parts.  First,  is  the  difference  between  the 
average  APUC  growth  post  Packard  reforms  (39%)  and  the  average  for  FY  1964-FY  1969 
(85%)  statistically  significant?  The  tests  used  found  this  difference  to  be  statistically 
significant  at  the  9%  level.10  It  is  worth  noting  these  reductions  probably  cannot  be  attributed 
only  to  the  policies  on  contract  type  that  Packard  instituted.  Four  of  the  20  programs  in  the 
data  set  for  FY  1964-FY  1969  used  TPP,  and  one  used  a  Firm  Fixed  Price  (FFP) 
development  contract.  The  average  APUC  growth  for  these  five  contracts  was  131%;  the 
average  cost  growth  for  the  remaining  FY  1961-FY  1969  programs  was  70%.*  11  TPP  and 
FFP  contracts  were  less  commonly  used  during  FY  1970-FY  1980,  but  three  of  the  MDAPs 


10  The  Mann-Whitney  U  test  rejected  the  null  hypothesis  (P  =  0.093)  that  the  samples  for  the  DSARC 
period  and  the  McNamara-Clifford  period  were  drawn  from  the  same  population,  (m  =  53,  ni  =  20,  U 
=  394).  A  two-tail  t-test  assuming  unequal  sample  variances  found  the  difference  in  the  means  to  be 
significant  (p  =  0.074).  The  Kolmogorov-Smirnov  (K-S)  test  showed  that  APUC  growth  estimates  for 
the  McNamara-Clifford  period  probably  are  not  normally  distributed.  The  result  of  the  t-test,  even  with 
the  correction  for  unequal  variances,  is  therefore  somewhat  suspect. 

11  For  further  discussion  of  TPP  and  FFP  development  contracts,  see  Tyson  et  al.  (1992,  Chapter  X); 
McNicol  (2004,  pp.  53,  57-59);  and  O’Neil  and  Porter  (2011,  p.  29-31). 
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that  passed  MS  ll/B  during  this  period  used  a  TPP  contract  and  one  used  an  FFP 
development  contract. 

Second,  are  the  differences  in  average  APUC  growth  for  the  three  periods  between 
McNamara-Clifford  and  AR  statistically  significant?  The  tests  used  did  not  reveal  any 
statistically  significant  differences  between  the  averages  of  APUC  growth  in  these  three 
periods.12  This  implies  that  the  lower  average  APUC  growth  (32%)  of  MDAPs  that  passed 
MS  II  during  the  DAB  years  (FY  1990-FY  1993),  for  example,  cannot  be  attributed 
confidently  to  the  full  implementation  of  the  DAB  in  1990,  because  a  change  of  this  size  has 
a  considerable  probability  of  occurring  by  chance. 

Third,  and  finally,  were  the  AR  years  associated  with  significantly  higher  average 
APUC  growth?  The  results  in  this  case  were  mixed.  One  test  indicated  that  average  APUC 
growth  over  the  AR  years  was  significantly  higher  than  it  was  in  FY  1990-FY  1993.  That 
result,  however,  was  not  confirmed  by  another  test.13  This  is  similar  to  the  result  found  in  P- 
5126  and  it  occurs  for  the  same  reason — the  variability  of  APUC  growth  in  the  AR  period 
was  too  large  for  the  differences  in  the  means  to  be  statistically  significant. 

The  Bayesian  analysis  presented  in  Appendix  C  of  McNicol  et  al.  (2016)  provides  a 
stronger  result  for  the  AR  years.  It  finds  clear  evidence  that  both  the  McNamara-Clifford 
period  (FY  1964-FY  1969)  and  the  AR  years  (FY  1994-FY  2000)  had  a  much  higher 
probability  of  high  cost  growth  than  did  the  bust  climate  portion  of  any  of  the  three 
intervening  periods  (DSARC,  Post-Carlucci  DSARC,  and  DAB). 

Returning  to  the  interpretation  of  Table  1  offered  above,  the  statistical  analysis  of 
average  APUC  growth  supports  two  of  the  three  points  offered  above — the  Packard  reforms 
did  reduce  APUC  growth  and  the  further  reforms  introduced  post-Packard  and  pre-AR  did 
not  yield  significant  further  reductions  in  APUC  growth.  The  results  on  the  third  point  are  not 
clear-cut.  The  statistical  tests  reported  above  do  not  support  attributing  the  high  mean 
APUC  growth  during  FY  1994-FY  2000  to  acquisition  reform,  but  the  Bayesian  analysis 
does  support  such  an  interpretation. 

Statistical  Analysis  of  the  Proportion  of  Extremely  High  APUC  Growth  Programs 

The  preceding  section  looked  for  effects  of  acquisition  policy  and  process  in 
differences  between  successive  periods  in  the  average  APUC  growth  of  MDAPs  that  passed 
MS  ll/B  during  them.  Although  reasonable,  framing  the  analysis  in  this  way  glosses  over  the 
possibility — explored  in  this  section — that  acquisition  policy  and  process  mainly  work  by 
influencing  the  proportion  of  MDAPs  that  experience  extremely  high  cost  growth. 

Some  relevant  data  are  provided  in  Table  2.  The  average  APUC  growth  figures  are 
the  same  as  those  presented  in  Table  1 .  In  addition,  Table  2  reports  the  number  of  MDAPs 


12  We  compared  the  three  periods  using  Analysis  of  Variance  (ANOVA).  ANOVA  failed  to  reject  the 
null  hypothesis  that  the  observations  in  the  three  periods  were  drawn  from  identical  normal 
populations.  The  K-S  test  found  it  highly  likely  that  the  samples  were  consistent  with  ANOVA’s 
assumptions. 

13  A  two-tail  t-test  assuming  unequal  variances  found  the  difference  to  be  significant.  (P  =  0.084.)  The 
K-S  test  rejected  the  null  hypothesis  that  the  observations  for  FY  1994-FY  2000  were  normally 
distributed.  A  Mann-Whitney  U  test  did  not  find  a  significant  difference  between  the  average  APUC 
growth  of  the  AR  years  and  that  for  the  period  FY  1990-FY  1993. 
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in  the  cohort  that  experienced  at  least  three  different  levels  of  APUC  growth — 50%,  100%, 
and  one  standard  deviation  (S)  above  the  sample  mean  (X).  The  sample  mean  is  57.4%  and 
the  standard  deviation  is  85.4%,  so  one  standard  deviation  beyond  the  mean  is  143%.  (X 
and  S  are  computed  for  the  bust  periods  only.)  In  what  follows,  MDAPs  in  the  last  of  the 
categories  will  be  called  “extremely  high  cost  growth”  programs.  These  are  arbitrary  breaks 
adopted  because  they  proved  to  be  useful.  Note  that  the  figures  for  the  number  of  systems 
in  the  right  tail  are  not  additive.  For  example,  of  the  20  MDAPs  that  entered  EMD  during  the 
period  FY  1 964-FY  1 969,  1 0  had  APUC  growth  of  at  least  50%.  Of  these  1 0,  six  had  APUC 
growth  of  more  than  100%,  and  of  the  six,  four  had  APUC  growth  of  more  than  143%. 

The  striking  feature  of  the  data  in  Table  2  is  the  paucity  of  extremely  high  cost  growth 
programs  after  the  introduction  of  the  Packard  reforms  in  1969  and  before  AR.  A  total  of  76 
programs  in  our  sample  passed  MS  II  during  the  18  years  of  the  DSARC,  Post-Carlucci 
DSARC,  and  DAB  periods  in  bust  funding  climates.  Only  one  of  these  has  an  estimated 
quantity  normalized  APUC  growth  from  the  MS  II  baseline  of  at  least  143%. 14  The  other  side 
of  this  coin  is  the  greater  frequency  of  extremely  high  cost  growth  systems  in  the 
McNamara-Clifford  years  and  during  the  AR  period.  Four  out  of  20  programs  of  the 
McNamara-Clifford  years  showed  extremely  high  cost  growth,  as  did  seven  out  of  27 
MDAPs  that  passed  MS  II  during  the  AR  years. 


Table  2.  Average  APUC  Growth  by  Acquisition  Regime  and  the  Number  of  High 
Cost  Growth  MDAPs  in  Each  Cohort,  Bust  Funding  Climates 


Acquisition  Regime 

Period 

(FY) 

Average 

APUC 

Growth* 

>  50% 

>  100% 

«/> 

+ 

IX 

Al 

McNamara-Clifford 

1964-1969 

85%  (20) 

10 

6 

4 

DSARC 

1970-1980 

39%  (53) 

19 

7 

0 

Post-Carlucci  DSARC 

1987-1989 

44%  (12) 

4 

3 

1 

DAB 

1990-1993 

32%  (11) 

5 

1 

0 

Acquisition  Reform 

1994-2000 

78%  (27) 

11 

7 

7 

DAB  post-AR 

2001-2002 

113%  (6) 

2 

1 

1 

Note.  Numbers  in  parentheses  are  the  number  of  observations  in  the  cell. 

*  Normalized  for  changes  in  quantity. 

Statistical  analysis  gives  substantially  the  conclusions  suggested  by  inspection  of  the 
data  in  Table  2: 

•  The  frequency  of  extremely  high  cost  growth  programs  was  significantly 
higher  in  the  McNamara-Clifford  years  than  in  the  DSARC  period. 


14  This  is  the  FGM-148A  Javelin.  Roland  also  had  a  very  high  APUC  growth  (308%)  but  was  placed 
on  the  cancelled  list.  Roland  was  developed  during  the  mid-1960s  by  a  French-German  consortium. 
In  1975,  the  U.S.  Army  decided  to  develop  and  procure  a  U.S.  version.  The  planned  procurement 
was  severely  reduced,  but  enough  was  acquired  to  equip  one  Army  National  Guard  battalion.  This 
does  not  fully  meet  the  definition  of  a  cancellation  but  was  judged  to  be  closer  to  a  cancellation  than 
to  a  truncation  of  the  program. 
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•  The  frequency  of  extremely  high  cost  growth  programs  also  was  significantly 
higher  during  the  AR  years  than  during  the  DSARC  period.15 

In  contrast  to  the  results  of  the  preceding  section,  both  the  McNamara-Clifford  period 
and  the  AR  period,  then,  stand  out  as  having  a  significantly  larger  proportion  of  extremely 
high  cost  growth  programs. 

Table  3  lists  the  extremely  high  cost  growth  systems.  Thirteen  of  the  14  passed  MS 
ll/B  during  bust  climates.  Helicopters  (2),  satellite  programs  (3),  and  launch  vehicles  (2)  are 
over-represented  but  do  not  dominate  the  list,  particularly  for  the  1960s. 


Table  3.  Extremely  High  Cost  Growth  Systems 


System  Name 

MS  ll/B  FY  APUC 

Bust  Climates 

AGM-69  Short  Range  Attack  Missile  (SFtAM) 

1967 

4.56 

MIM-23  Hawk  (Improved  Hawk) 

1965 

2.07 

Versatile  Avionics  Shop  Test  (VAST) 

1968 

1.83 

M47  Dragon  Guided  Missile 

1966 

1.72 

FGM-148A  Javelin  Advanced  Anti-Tank  Weapon  System 

1989 

1.59 

Space  Based  IR  Sensor  (SBIRS)  High 

1997 

3.90 

Evolved  Expendable  Launch  Vehicle  (EELV) 

1998 

3.42 

Global  Broadcast  Service  (GBS) 

1998 

2.60 

Guided  Multiple  Launch  Rocket  System  (GMLRS) 

1998 

2.15 

H-1  Upgrades 

1996 

1.97 

CH-47F  (Improved  Cargo  Helicopter) 

1998 

1.81 

Patriot  Advanced  Capability-3  (PAC-3) 

1994 

1.49 

Advanced  Extremely  High  Frequency  (AEHF)  Satellite 

2001 

4.78 

Boom  Climates 

Titan  IV  Expendable  Launch  Vehicle  (ELV) 

1985 

1.49 

We  also  explored  whether  the  proportions  of  systems  with  cost  growth  of  at  least 
50%  or  100%  might  show  the  same  pattern  across  acquisition  periods  as  the  extremely  high 
cost  growth  systems.  Analyses  parallel  with  those  just  described,  with  observations  of  at 
least  50%  APUC  growth  and  100%  APUC  growth  showing  no  significant  differences  across 
the  acquisition  periods. 

Appendix  D  of  McNicol  et  al.  (2016)  presents  results  obtained  from  a  technique 
(quantile  regression)  that  compares  the  APUC  growth  distributions  across  acquisition 
regimes  at  several  points.  The  comparison  reported  used  deciles.  The  results  were 


15  These  statements  are  based  on  results  for  Fisher’s  Exact  Tests:  (1)  p  =  0.004  in  the  comparison  of 
McNamara-Clifford  to  the  DSARC  years,  and  (2)  p  <  0.001  for  the  comparison  of  FY  1994-FY  2000 
with  the  DSARC  years. 
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consistent  with  those  stated  above  in  two  respects:  (1 )  There  were  no  significant  differences 
across  the  six  acquisition  periods  in  the  central  portions  of  the  distribution  (4th  through  the 
7th  deciles),  and  (2)  the  McNamara-Clifford  and  AR  periods  had  significantly  fatter  right  tails. 
It  also  is  interesting  to  note  that  there  is  some  evidence  that  the  left  tails  of  these  two 
periods  were  somewhat  fatter  than  those  of  other  periods;  that  is,  McNamara-Clifford  had 
higher  highs  and  perhaps  higher  lows. 

Finally,  we  considered  the  pattern  in  average  APUC  growth  across  the  six  acquisition 
periods  if  the  13  extremely  high  cost  growth  programs  are  removed.  The  means  of  the 
truncated  distributions  are  presented  in  Table  4.  Pair-wise  tests  found  the  average  APUC 
growth  for  the  AR  years  (without  the  extremely  high  cost  growth  systems)  to  be  significantly 
lower  than  the  averages  for  the  McNamara-Clifford  and  DSARC  periods.  None  of  the  other 
differences  was  statistically  significant  and  a  test  of  the  table  as  a  whole  did  not  reveal 
significant  differences.16  It  appears,  then,  that  the  significant  differences  in  average  APUC 
growth  reported  in  the  previous  section  (Statistical  Analysis  of  Average  Cost  Growth)  stem 
from  the  significantly  higher  proportion  of  extremely  high  cost  growth  systems  during  the 
McNamara-Clifford  and  AR  periods. 

Table  4.  Average  APUC  Growth  by  Acquisition  Regime  for  Bust  Funding 
Climates,  Excluding  Extremely  High  Cost  Growth  Programs 


Acquisition  Regime 

Period  (FY) 

Average  APUC 
Growth* 

McNamara-Clifford 

1964-1969 

0.43  (16) 

DSARC 

1970-1980 

0.39  (53) 

Post  Carlucci  DSARC 

1987-1989 

0  34  (11) 

DAB 

1990-1993 

0  32  (11) 

Acquisition  Reform 

1994-2000 

0.18(20) 

DAB  post-AR 

2001-2002 

0.40  (5) 

Note.  Numbers  in  parentheses  are  the  number  of  observations  in  the  cell. 

*  Normalized  for  changes  in  quantity. 

Appendix  E  of  McNicol  et  al.  (2016)  presents  an  analysis  of  the  boom  case  that 
parallels  that  of  this  and  the  preceding  section  for  the  bust  case.  There  was  no  indication  of 
significant  association  between  acquisition  period  and  average  APUC  growth  and  no 
indication  of  statistically  significant  differences  across  the  acquisition  regimes  in  the  boom 
periods  with  the  proportion  of  MDAPs  in  the  right  tail  of  the  distributions. 

Interpretation  of  the  Statistical  Results 

The  conclusions  of  the  preceding  section  add  a  level  of  detail  to  the  interpretation  of 
the  APUC  growth  data  offered  in  the  earlier  section  titled  Statistical  Analysis  of  Average 


16  Two-tail  t-test  of  the  differences  of  the  means  of  two  independent  samples.  ANOVA  for  the  table  as 
a  whole  yielded  P  =  0.45.  K-S  found  four  of  the  distributions  to  be  normal.  The  exceptions  were  those 
for  FY  1 964-FY  1 969,  which  K-S  found  to  only  marginally  satisfy  the  test  for  normality,  and  FY  2001- 
FY  2002,  which  had  too  few  data  points  to  test. 
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Cost  Growth.  Packard’s  radically  new  acquisition  phases  and  his  more  highly  structured 
process  were  almost  completely  successful  in  preventing  instances  of  extremely  high  cost 
growth  and,  for  this  reason,  significantly  reduced  average  APUC  growth.  The  relaxation  of 
OSD  oversight  of  MDAPs  during  the  AR  era  saw  a  return  of  a  significant  number  of 
extremely  high  cost  growth  systems  and,  for  that  reason,  average  APUC  growth  returned  to 
nearly  its  1960s  level.  In  sum,  the  Packard  reforms  of  late  FY  1969  worked  well  in 
essentially  eliminating  instances  of  extremely  high  cost  growth  and  in  that  way  reduced 
average  APUC  growth;  they  were  not  significantly  improved  upon  in  this  respect  through  the 
early  2000s;  and  the  relaxation  of  OSD-level  oversight  of  the  AR  years  was  associated  with 
a  significant  number  of  extremely  high  cost  growth  programs  and,  therefore,  of  higher 
average  APUC  growth. 

The  DAB  process  is  a  mechanism  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  can  use  to  bring  MDAPs  into  conformance  with  acquisition  policy 
at  MS  ll/B.  Among  other  things,  programs  should  have  use  the  appropriate  contracting 
mechanism,  should  have  a  sound  test  plan,  should  not  proceed  until  the  technologies  to  be 
employed  are  reasonably  mature,  should  rest  on  realistic  programmatic  assumptions,  and 
should  be  fully  funded  to  a  realistic  cost  estimate.  It  is  not  surprising,  then,  to  find  that 
(except  in  the  AR  years  when  OSD-level  oversight  was  relaxed)  the  DSARC  process  and  its 
successor,  the  DAB  process,  largely  eliminated  instances  of  extreme  cost  growth.  This 
might  be  due  to  direct  OSD-level  modification  of  particular  MDAPs.  Alternatively,  the 
certainty  of  reviews  by  the  DSARC/DAB  might  have  prompted  the  Services  to  avoid  in  the 
programs  they  proposed  the  characteristics  that  cause  high  cost  growth.  The  best  way  to 
gain  a  deeper  insight  into  the  matter  probably  is  to  compare  closely  the  AR  period  with  the 
DSARC  period  and  to  examine  the  extremely  high  cost  growth  programs. 

It  is  surprising  that  the  statistically  significant  differences  are  found  only  for  the 
extremely  high  cost  growth  systems.  The  description  of  the  process  certainly  suggests  that  it 
also  should  have  an  effect  on  programs  with  smaller  but  still  very  substantial  cost  growth. 
This  finding,  however,  does  not  necessarily  imply  that  the  OSD-level  process  has  no  effect. 
Instead,  the  statistical  finding  as  such  is  that  the  fairly  rudimentary  OSD-level  process  of  the 
McNamara-Clifford  years  did  as  well  as  its  more  elaborate  successors  except  on  extremely 
high  cost  growth  systems. 

It  is,  finally,  important  to  note  that  this  paper  has  been  concerned  almost  entirely  with 
cost  growth  of  MDAPs  that  passed  MS  ll/B  in  bust  periods.  A  complete  summary  also  would 
need  to  take  into  account  parallel  analyses  for  the  boom  periods  and  the  comparisons  of 
cost  growth  in  bust  and  boom  periods  for  a  given  acquisition  regime.  That  task,  however,  is 
postponed  to  a  subsequent  study. 
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Background 


■  This  briefing  is  drawn  from: 

David  McNicol,  David  Tate,  Sarah  Burns,  and  Linda  Wu,  “Further  Evidence 
on  the  Effect  of  Acquisition  Regime  on  Cost  Growth,”  IDA  Paper  P-5330 
(Draft  Final),  April  2016. 

■  That  paper  extends  results  presented  in: 

David  McNicol  and  Linda  Wu,  “Evidence  on  the  Effect  of  DoD  Acquisition 
Policy  and  Process  and  Funding  Climate  on  Cost  Growth  of  Major  Defense 
Acquisitions  Programs,”  IDA  Paper  P-5126,  September  2014. 

■  These  results  were  presented  at  last  year’s  NPS  Acquisition 
Research  Symposium. 

■  Both  papers  were  sponsored  by  the  Director,  Performance 
Assessments  and  Root  Cause  Analyses. 

■  The  project  attempts  a  quantitative  assessment  of  the  effects  of 
changes  in  acquisition  policy  and  process  on  MDAP  outcomes. 
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1st  Observation: 

(a)  The  Packard  reforms  of  1969  significantly  reduced  average  APUC 
growth,  (b)  Additional  reforms  over  about  the  next  25  years  did  not 
further  reduce  APUC  growth,  (c)  During  the  AR  years,  APUC  growth 
returned  to  nearly  its  1960  level. 


Acquisition  Regime 

Average 

APUC 

Growth* 

McNamara-Clifford 

|  1964-1969 

85%  (20) 

Defense  Systems  Acquisition 
Review  Council  (DSARC) 

1970-1980 

39%  (53) 

Post  Carlucci  Initiatives  DSARC 

1987-1989 

44%  (12) 

Defense  Acquisition  Board  (DAB) 

1990-1993 

32%  (11) 

Acquisition  Reform  (AR) 

1994-2000 

78%  (27) 

DAB  post  AR 

2001-2002 

113%  (6) 

a 


c 


b 


*  Normalized  for  changes  in  quantity. 
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2nd  Observation: 

(a)  MDAPs  that  passed  MS  ll/B  during  McNamara-Clifford  and  during  AR 
had  significantly  higher  proportions  of  programs  that  experienced 
extremely  high  APUC  growth.*  (b)  The  proportions  across  acquisition 
regimes  were  not  significantly  different  at  50  percent  and  100  percent. 


Acquisition  Regime 

Average  APUC 
Growth** 

>  50% 

>  1 00% 

>x  +  s 

McNamara-Clifford 

1964-1969 

85%  (20) 

10 

6 

4 

DSARC 

1970-1980 

39%  (53) 

19 

7 

0 

Post-Carlucci  DSARC 

1987-1989 

44%  (12) 

4 

3 

1 

DAB 

1990-1993 

32%  (11) 

5 

1 

0 

Acquisition  Reform 

1994-2000 

78%  (27) 

11 

7 

7 

DAB  post-AR 

2001-2002 

113%  (6) 

2 

1 

1 

*  Extremely  high  APUC  growth:  at  least  one  standard  deviation  (S)  above  the  sample  mean  (X)  for  bust 
periods.  X"+  S  =  143%. 

**  Normalized  for  changes  in  quantity. 
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3rd  Observation: 

There  is  no  statistical  significant  association  between 
acquisition  regime  and  average  APUC  growth  in  bust  funding 
climates  when  extremely  high  cost  growth  observations*  are 
excluded. 


Acquisition  Regime 


McNamara-Clifford 


DSARC 


Post  Carlucci  Initiatives  DSARC 


Defense  Acquisition  Board  (DAB) 


Acquisition  Reform  (AR) 


DAB  post  AR 


Average 

APUC 

Growth** 

1964-1969 

43% 

1970-1980 

39% 

1987-1989 

34% 

1990-1993 

32% 

1994-2000 

18% 

2001-2002 

40% 

*  APUC  growth  greater  than  >  X  +  S  =  143%. 
**  Normalized  for  changes  in  quantity. 
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Cautions 


■  The  crucial  statistically  visible  effects  of  acquisition  regimes  are  on 
the  relative  frequency  of  extremely  high  cost  growth  programs  (2nd 
Observation). 

■  Apart  from  the  effects  on  extremely  high  cost  growth  programs, 
there  is  not  a  statistically  significant  association  between  changes  in 
acquisition  regime  and  average  APUC  growth  of  programs  that 
passed  MS  ll/B  during  the  regime.  (3rd  Observation) 

■  This  does  not  imply  that  OSD-level  oversight  of  MDAPs  has  no 
effect.  It  does  imply,  however,  that  changes  in  OSD-level  acquisition 
policy  and  process  relative  to  McNamara-Clifford  did  not  make 
APUC  growth  either  better  or  worse  for  MDAPs  that  passed  MS  ll/B 
during  bust  funding  periods. 
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Some  Opinions  Suggested  by  the  Observations 


■  It  seems  unlikely  that  further  changes  in  acquisition  policy  or  process 
will  have  a  major  effect  on  cost  growth. 

■  The  relevant  context  for  understanding  APUC  growth  is  the  interface 
between  the  acquisition  process  and  the  program/budget  process. 

■  The  root  causes  of  cost  growth  arguably  lie  in  a  mismatch  among 
mission  and  functions,  force  structure,  and  function. 

■  When  something  must  give,  something  does  give. 

■  Extremely  optimistic  programmatic  or  cost  assumptions  may  appear  to  be 
the  least  bad  option. 

■  The  underlying  cause  of  persistent  high  APUC  growth  is  not  a  deeply 
established  culture  of  the  DoD  acquisition  organizations  and  their 
professional  employees. 
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In  progress 


■  The  Effect  of  Post  MS  ll/B  Funding  Climate  on  Cost  Growth. 

■  Further  Evidence  on  the  Effect  of  Acquisition  Regime  and  Funding 
Climate  on  RDT&E  Growth  (parallel  to  P-5330’s  analysis  of  APUC 
growth). 

■  Further  Evidence  on  the  Effect  of  Acquisition  Regime  and  Funding 
Climate  on  Truncations  and  Cancellations  (IDA  P-5218) 


■  IDA  publication  that  consolidates  this  work  and  P-5126,  P-5218,  and 


P-5330. 
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Backup 
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The  Outcome  Metric — Growth  in  Average 

Procurement  Unit  Cost  (APUC) 


■  The  research  used  APUC  growth  for  190  Major  Defense  Acquisition 
Programs  (MDAPs)  that  passed  MS  ll/B  during  FY  1970-FY  2007. 

■  Each  of  these  MDAPs  went  into  production. 

■  There  are  no  cancelled  programs  in  the  sample. 

■  APUC  growth  is  measured  from  the  MS  ll/B  baseline  and 
normalized  to  the  MS  ll/B  total  quantity  acquired. 

■  APUC  growth  over  the  entire  acquisition  cycle  is  associated  with  the 
Fiscal  Year  in  which  the  MDAP  passed  MS  ll/B;  for  example: 

■  APUC  growth  for  the  F-22  over  FY  1991-FY  2006  is  assigned  to  FY 
1991 ,  the  year  in  which  the  F-22  passed  MS  II. 

■  The  average  APUC  growth  for  FY  1987-FY  1993  is  the  average 
quantity  normalized  PAUC  growth  of  all  MDAPs  that  passed  MS  II 
during  those  years. 
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Budget  Authority  Appropriated  for  Procurement, 

1960-1982* 


200 

180 

160 

140 

120 

100 

80 

60 

40 

20 

0 


1st  Bust-Boom  Cycle 


2nd  Bust-Boom  Cycle 


I960  1970  1980  1990  2000  2010 

*  In  Billions  of  Constant  FY  2015  Dollars. 

Source :  “National  Defense  Budget  Estimates  for  FY  2014,”  Office  of  the  Under 
Secretary  of  Defense  (Comptroller),  May  2013,  Table  6-8,  142-148. 
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Selection  of  Break  Points  between  Funding  Climates 

■  This  research  used  only  two  funding  climate  categories — Relatively 
Constrained  and  Relatively  Accommodating. 

■  Our  touchstone  in  selection  of  break  points  was  major  shifts  in  the 
expectation  about  future  funding  of  senior  DoD  decision  makers. 

■  We  used  three  events  to  identify  the  break  points  between  funding  climates: 

■  The  invasion  of  Afghanistan  by  the  USSR  in  late  December  1979; 

■  The  passage  of  the  Gramm-Rudman-Hollings  Act  in  December  1985;  and 

■  The  terrorist  attack  on  the  U.S.  on  Sept.  1 1 , 2001 . 

■  Senior  decision  makers  could  reasonably  expect  each  of  these  events  to 
result  in  major  and  sustained  changes  in  the  defense  funding  climate. 

■  After  examining  contemporary  policy  statements  and  events,  we  selected: 

■  FY  1 981  as  the  first  year  of  the  Carter-Reagan  buildup; 

■  FY  1 986  as  the  final  year  of  the  Carter-Reagan  buildup;  and 

■  FY  2003  as  the  first  year  of  the  post-9/1 1  defense  buildup. 
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Are  there  Other  Factors  that  Must  Be  Considered? 


■  The  statistical  examinations  consider  only  two  factors  bearing  on 
MDAP  outcomes — acquisition  regime  and  funding  climate. 

■  The  results  are  not  compromised  by  the  omission  of  factors  that  are 
uncorrelated  with  either  acquisition  regime  or  funding  climate — e.g., 
a  tendency  towards  increased  complexity  in  MDAPs  overtime. 

■  The  results  for  these  two  factors  would  be  compromised  by 
omission  of  factors  important  to  cost  growth  that  are  also  correlated 
with  acquisition  regime  or  funding  climate. 


The  results  sought  are  not  directly  about  the  causes  of  cost  growth, 
but,  rather,  when  and  where  cost  growth  occurs: 

■  When,  in  terms  of  chronology  and  of  the  point  in  the  acquisition  cycle 
at  which  the  main  seeds  of  cost  growth  are  sown;  and 

■  Where,  in  terms  of  which  bureaucratic  processes  are  implicated. 
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Methodology — Funding  Climate 


Acquisition  Regime 


^  A 

•  *  ^  ^ 

^  ^ 


McNamara-Clifford 


Operated  only  in  a  bust  funding  climate 


Defense  Systems 
Acquisition  Review 
Council  (DSARC) 


Post  Carlucci  Initiatives 
DSARC 


1970-1980 


1987-1989 


N  =  53 


N  =  12 


1981-1982 


1983-1986 


N  =  9 


N  =  34 


Defense  Acquisition 
Board  (DAB) 


Operated  only  in  a  bust  funding  climate 


Acquisition  Reform  (AR) 


Operated  only  in  a  bust  funding  climate 


DAB  post  AR 


2001-2002 


N  =  6 


2003-2007 


N  =  17 


Three  acquisition  regimes  operated  in  both  funding  climates. 
These  provide  a  way  to  gauge  the  effect  of  funding  climate, 

given  acquisition  regime. 
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Sample  Probability  of  Cancellation  by  PAUC 

Growth  Category 


Average  PAUC 
Growth 

25  MDAPs  that 
were  Cancelled 

151  MDAPs  that 
were  not  Cancelled 

Sample 
Probability  of 
Cancellation 

>  50% 

40%  (10) 

26%  (40) 

20%  (10  of  50) 

>  30%  but  <30% 

0(0) 

11%  (17) 

0  (0  of  17) 

0  to  30% 

40%  (10) 

46%  (64) 

14%  (10  of  74) 

<0 

20%  (5) 

19%  (30) 

1 5%  (5  of  35) 

Total 

100%  (25) 

100%  (151) 

14%  (25  of  176) 

Source:  IDA  P-5218,  pp.  9-10. 


Contrary  to  the  conventional  wisdom,  the  probability  that  an  MDAP 
will  be  cancelled  is  not  strongly  dependent  on  the  level  of  cost 

growth  it  experiences. 
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Methodology — Acquisition  Regime 


Acquisition  Regime 


McNamara-Clifford 


Defense  Systems 
Acquisition  Review 
Council  (DSARC) 


Post  Carlucci 
Initiatives  DSARC 


Defense  Acquisition 
Board  (DAB) 


Acquisition  Reform 
(AR) 


DAB  post  AR 


1964-1969 

N  =  20 

1970-1980 

N  =  53 

1987-1989 

N  =  12 

1990-1993 

N  =  11 

1994-2000 

N  =  27 

2001-2002 

N  =  6 

*  ^  ^ 

» -9* 


not  observed 


1981-1982 


1983-1986 


N  =  9 


N  =  34 


no  observations 


no  observations 


2003-2007 


N  =  17 


Each  funding  climate  is  analyzed  separately.  All  six  acquisition  regimes  operated  during 
periods  of  bust  funding;  only  three  of  the  regimes  operated  during  boom  periods.  We  looked 
at  both  average  APUC  growth  and  the  proportion  of  MDAPs  with  extremely  high  APUC 
growth. 
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Average  Annual  MDAP  Cancellation  Rates  in  Two 

Funding  Climates 


Acquisition 

Regime 

Sharply  Decreasing 
Funding 

Stable  or  Increasing 
Funding 

Period 

Cancellation 

Rate 

Period 

Cancellation 

Rate 

DSARC 

1970-1982 

0.5/yr. 

Post-Carlucci 

DSARC 

1986-1989 

2.3/yr. 

1983-1985 

0.3/yr. 

DAB 

1990-1993 

3.3/yr. 

AR 

1994-2000 

0.4/yr. 

DAB  post  AR 

2009-2012 

2.8/yr. 

2001-2008 

1 .3/yr. 

Annual  average  cancellation  rates  are  much  higher  when  procurement  funding 
is  sharply  decreasing  than  when  it  is  stable  or  increasing. 
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Average  Cancellation  Rates  for  Cohorts,  by  Acquisition 

Regime  and  Funding  Climate 


Acquisition 

Regime 


DSARC 


Post-Carlucci 

DSARC 


DAB 


AR 


DAB  post  AR 


■  *  ^ 

&  ^  * 


Period 


1970-1980 


1987-1989 


1990-1993 


1994-2000 


2001-2002 


Cancellation 

Rate 


14%  (10/74) 


42%  (11/26) 


Period 


1981-1982 


1983-1986 


Cancellation 

Rate 


24%  (4/17) 


12%  (6/51) 


7%  (1/15) 


14%  (6/43) 


27%  (4/15) 


Note:  Number  of  cancellations  and  cohort  size  in  parentheses. 

The  high  cancellation  rate  of  the  1987-1989  cohort  probably  reflects  a 
situation  unique  to  the  2nd  Reagan  Administration.  The  DAB  post  AR  also  had 
a  comparatively  high  cancellation  rate  for  both  funding  climates.  Otherwise, 
there  does  not  seem  to  be  an  association  between  cohort  cancellation  rates, 
acquisition  regime,  and  funding  climate. 
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Bias  in  the  MS  ll/B  PAUC  Estimate  and  the  Frequency  of 
PAUC  Growth  >  50  Percent  and  Negative  PAUC  Growth 


Relatively  Accommodating 
Funding  Climate 


Relatively  Constrained 
Funding  Climate 
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